Focused on the problems of multiple expression variations, multiple pose variations as well as varying-degree missing face point cloud data in Three-Dimensional (3D) faces, 3D point cloud face data was exploratively applied to PointNet series classification networks, and the recognition results were compared and analyzed, then a new network framework named HFN (Hierarchical Feature Network) was proposed. First, the point cloud with fixed points was randomly sampled after data preprocessing. Second, the point fixed point cloud was input into SA (Set Abstraction) module in order to obtain the centroid points and neighborhood points of the local areas, and extract the features of the local areas, then the point cloud spatial structural features extracted from DSA (Directional Spatial Aggregation) module based on multi-directional convolution were mosaicked. Finally, the full connection layer was used to perform the classification of 3D faces, so as to realize the 3D face recognition. The results on CASIA database show that the average recognition rate of the proposed method is 96.34%, which is better than those of classification networks such as PointNet, PointNet++, PointCNN and Spatial Aggregation Net (SAN).